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Executive Summary 


A research consortium of scientists and engineers from California State University Long Beach 
(CSULB), San Jose State University Foundation (SJSUF), California State University Northridge 
(CSUN), Purdue University, and The Boeing Company was assembled to evaluate the impact of 
changes in roles and responsibilities and new automated technologies, being introduced in the Next 
Generation Air Transportation System (NextGen), on operator situation awareness (SA) and 
workload. To meet these goals, consortium members performed systems analyses of NextGen 
concepts and airspace scenarios, and concurrently evaluated SA, workload, and performance 
measures to assess their appropriateness for evaluations of NextGen concepts and tools. The 
following is a summary of our activities and accomplishments that were supported by the NASA 
Research Announcement (NRA). 

• Distributed Simulation. Concept and metric evaluation required that we develop a 
distributed simulation network in which each laboratory participated in large-scale 
simulations of NextGen concepts. With this distributed-simulation foundation established, 
two distributed-simulation demonstrations and two distributed-simulation experiments were 
completed in year 3 of the NRA. 

• Metric Development. Based on our review of SA and SA measurement, we determined that 
probe techniques are the most promising measures for evaluating the impact of NextGen 
changes in roles and responsibilities on operator SA and workload. The probe techniques 
have been shown to have relatively good psychometric properties, especially predictive 
validity and diagnosticity. We modified the method developed by Durso (e.g., Durso and 
Dattel, 2004) by automating SA probing, improving on the method of presenting probe 
questions and collecting operator responses, working toward a standard set of probe 
questions (or probe question categories) and adding workload queries. Several simulations 
were run on both pilots and air traffic controllers (ATCs). These provided evidence for the 
validity, sensitivity and diagnosticity of the method. 

• Systems Analysis. System analyses of potential NextGen concepts of operation were also 
reviewed. We determined that these concepts could be distributed along two axes: the 
degree to which responsibility for separation assurance and collision avoidance is assigned 
to the controllers versus the pilots, and the degree to which automation augments or replaces 
controller and pilot functions. Interviews with air traffic controllers and commercial pilots 
for task goals information requirements were also done. ATCs were also interviewed about 
training requirements for NextGen operators. ATCs responded that the next generation of 
air traffic controllers would not benefit much from previous experience with computers and 
games because the perceptual motor skills are not the most important skill involved in air 
traffic management. 

• Part-Task Simulations . A series of part-task simulations and experiments were conducted 
within each consortium organization. These were focused on conflict detection and 
resolution in NextGen, development of metrics for SA and workload in NextGen, 
application of the metrics to NextGen airspace environments, and the evaluation of 
NextGen concepts of operation for separation assurance. These were followed by a series of 
distributed air- ground simulations that involved simulation roles (flight decks, controllers, 
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pseudo-aircraft, etc.) being played by two or more member organizations. Part-task- 
simulation experiments directed at metric development established the validity, sensitivity 
and diagnosticity of the modified online probe technique. Experiments on pilot conflict 
resolution showed that pilots were not comfortable accepting automated conflict resolutions 
and desired to communicate with ATCs more than is anticipated in NextGen. 

• Large-Scale Simulations . Our distributed simulations evaluated plausible NextGen concepts 
of operations, and cockpit automation tools for traffic separation, weather avoidance, 
merging and spacing, and communications. These projects simulated sectors in Kansas City 
Center (ZKC) and Indianapolis Center (ZID), with participant pilots flying en route, arrival 
and approaches to Louisville International Airport (SDF), while avoiding weather and 
managing spacing. Three concepts of operation were evaluated that allocated primary 
responsibility for traffic separation between pilots, ATCs and automation. We determined 
that SA was highest when pilots were responsible for traffic separation. Workload was 
unaffected by adding responsibility for traffic separation. Workload was only affected by 
task demands related to flight phase. Weather perturbations did create situations in which 
aircraft spacing was negatively affected regardless of whether the pilot faithfully followed 
the leading aircraft around weather. 

As a result of this NRA, we have gained a greater understanding of SA and its measurement, and 
have shared our knowledge with the scientific community: During the NRA, we published 20 
technical papers, made 21 presentations at conventions, presented 6 briefings to the National 
Aeronautics and Space Administration (NASA), and prepared 5 unpublished reports. Additional 
technical papers based on the NRA work were also published after the funding period. Through the 
success of our distributed- simulation network, we have shown how to create a flexible and cost- 
effective test bed for simulations and research involving current and future airspaces. This network 
provides a mechanism for consortium members, colleagues, and students to pursue research on 
other topics in air traffic management and aviation, thus enabling them to make greater 
contributions to the field. 
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1. Introduction 

1.1 Objectives 

Introducing new air traffic management (ATM) concepts, Automated Separation Assurance Systems 
(AS AS), and other automation technologies proposed by Next Generation Air Transportation System 
(NextGen) can significantly impact workload and situation awareness (SA) for operators in future 
airspace systems. Successful implementation of these solutions, for achieving Joint Planning and 
Development Office (JPDO) goals, demands that we identify the information required of each 
operator in performing his/her new role, and the impact of new technologies on operator SA and 
workload. As a result, current metrics need to be revised or new measurement techniques need to be 
developed to ensure that they are reliable, valid, and sensitive to the changes in operator SA and 
workload brought about by NextGen innovations. The objectives established for our NASA Research 
Announcement (NR A) are as follows: 

• Develop methods for quantifying required and actual operator SA, workload, and 
performance in relation to: 

- Operator (flight crew and controller) role in Automated Separation Assurance 
Systems 

- The impact of these factors on system safety and performance under NextGen 

• Characterize individual and shared SA in NextGen environments. 

• Support program capability for human-in-the-loop simulations of NextGen concepts and 
evaluations of SA, workload, and performance. 

1.2 Approach 

A research consortium of scientists and engineers from California State University Long Beach 
(CSULB), San Jose State University Foundation (SJSUF), California State University Northridge 
(CSUN), Purdue University, and The Boeing Company was assembled. Our approach to the 
development and evaluation of SA, workload, and performance was characterized by systems 
analysis (conducted by systems engineers and operations experts, with inputs from NASA) of 
NextGen concepts and airspace scenarios. Concurrently, metric evaluations of existing measures of 
SA, workload, and performance were conducted to assess their ability to predict SA in NextGen 
environments. These parallel approaches drove the design of simulations to evaluate candidate 
measures for their ability to reflect changes in operator SA, workload, and performance that might be 
induced by NextGen concepts. 

Our multidisciplinary team of universities and private industry established the technological 
capacity to conduct networked, human-in-the-loop simulations for assessing candidate SA metrics, 
workload, and performance for individual pilots and air traffic controllers (ATC), pilot-controller 
teams, and pilot-controller interactions with AS AS. All members of the consortium have extensive 
knowledge and experience in the areas of human performance and ATM operations. Table 1 
describes the member organizations in terms of the institution, key personnel, and the role of each 
team member in the NRA. 
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Table 1. Consortium Organizations and Descriptions of 
Principal Investigators (PI) and Co-Investigators (Cl) 


California State University, Long Beach; Center for Human Factors in 
Advanced Aeronautic Technologies (CHAAT) 

• Thomas Strybel, PI; Professor of Psychology and Director of 
CHAAT 

• Kim-Phuong L. Vu, Co-1; Associate Professor of Psychology and Co- 
Director of CHAAT 


San Jose State University Foundation 

• Vernol Battiste, Co-1; Senior Research Associate for the FDDRL at 
NASA Ames Research Center 

• Arik-Quang V. Dao, Co-1; Research Associate for the FDDRL 


Purdue University, Human Integrated Systems Engineering Laboratory 
(HSEL) 

• Steven Landry, Co-1; Assistant Professor of Industrial Engineering 
and Director of HISEL 


California State University, North ridge; Systems Engineering Research 
Laboratory (SERL) 

• Nhut Ho, Co-1; Associate Professor of Mechanical Engineering and 
Director of SERL 


The Boeing Company 

• John P. Dwyer, Co-1; Technical Fellow, Human Systems Integration 


By taking advantage of the unique strengths and capabilities of each member the consortium, we made 
significant progress in achieving the goals of the NRA. Our objectives were accomplished through: 

• Part task, in-house simulations conducted individually at consortium laboratories 

• Distributed simulations involving two or more consortium organizations 

• Laboratory experiments conducted individually within the consortium organizations 

• Systems analyses and concept papers developed by one or more Co-Investigator. 

Our accomplishments with respect to the objectives of this NRA are detailed in the subsequent pages. 
They have been organized into three major sections. Section 2 summarizes the methods we developed 
for distributed simulation capability, and SA and workload measurement. Section 3 summarizes 
systems analyses we conducted of operator roles and responsibilities in NextGen, information 
requirements and implications of NextGen concepts on operator training. Section 4 summarizes the 
results obtained from application of the methods, the knowledge we have obtained regarding how to 
measure SA, workload, and performance in NextGen air traffic environments, and evaluations of 
NextGen concepts in terms of changes in operator roles and their effect on operator SA, workload, 
and performance. 
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2. Method Development 

2.1 Distributed Simulation Capability 

Our consortium members have established a simulation network infrastructure that permits 
interaction between controllers, pilots, and experimenters over the Internet. This capability is made 
possible through the implementation of software donated to the consortium members by NASA Ames 
Research Center: Aeronautical Data Link and Radar Simulator (ADRS), Multi Aircraft Control 
System (MACS), Four Dimensional Cockpit Situation Display (4D-CSD), and Distributed 
Air/Ground Voice communication. Taking advantage of both the existing infrastructure at each 
consortium site and the adaptability of NASA Ames’ simulation software, we created an extremely 
flexible simulation network. Building on NASA’s own plan for a versatile simulation environment, 
our distributed simulation capability enabled us to readily host large simulations that can vary the 
number of self-separation-capable aircraft, simulate current day, NextGen, or mixed-equipage 
airspace environments, and assess how different levels of automation associated with different 
NextGen concepts of operations affect operator and system performance. 

The distributed simulation capability was successfully demonstrated in 2008. First, we established 
connectivity and ensured adequate bandwidth at each facility. Then we installed simulation software 
and trained personnel at each facility on simulation software and procedures, thus expanding the 
capacity for airspace simulations in one facility (CHAAT at CSULB) and creating two additional 
simulation facilities in consortium laboratories (SERL at CSUN and HISEL at Purdue). Once 
operational, FDDRL engineers improved the distributed simulation network by creating a capability 
for remote startup of all simulation stations to ensure that all stations at each facility were 
appropriately connected and configured before the start of a simulation run. With this distributed 
simulation foundation established, two distributed- simulation demonstrations and two distributed- 
simulation experiments were completed in year 3 of the NRA. Table 2 shows the distributed 
simulation configuration for one of the simulation experiments, the Situation Awareness in Trajectory 
Oriented Operations with Weather II (SAWTOOth II) simulation that was conducted between August 
and September 2009, described in Section 4.2. This simulation involved over 30 workstations at four 
consortium laboratories. 


Table 2. Distributed Simulation Components for the Situation Awareness 
and Workload in Trajectory Oriented Operations with Weather Simulation* 


FDDRL 
(NASA Ames 
Research Center) 

CSAAT 

(Calif. State Univ., Long 
Beach) 

HISEL 

(Purdue Univ.) 

SERL 

(Calif. State Univ., 
Northridge) 

• 6 participant pilot 

• 2 participant ATC 

• 1 ADRS station 

• 2 single 

stations 

stations 

• 2 single pseudopilot 

pseudopilot 

• 5 single pseudopilot 

• 2 ghost/confederate 

stations 

stations 

stations 

• 1 777 simulator 

• 1 eye-tracking 
station 

• 1 Simulation 
Manager station 

• 1 Voice Server 

• 1 ADRS station 

ATC stations 

• 8 multi-pilot stations 

• 2-4 probe stations 

• 1-2 ADRS stations 


• 1 ADRS station 


* See Section 4.2. 
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In summary, our development of a distributed simulation capability not only created a mechanism for 
running large-scale simulations of most NextGen operator roles, but also expanded the number of 
universities capable of performing NextGen airspace simulations. 

2.2 Metric Development 

NextGen innovations must be developed using a system-comprehensive, user-centered strategy that 
promises continuing improvement in efficiency and maintenance— and even enhancement— of safety. 
However, NextGen automation solutions may alter operator SA and workload in negative ways, and 
it is important that valid and reliable measures be available for evaluating SA, workload, and 
performance in NextGen. As part of the NRA, we evaluated existing metrics, and developed new 
techniques, for quantifying operator SA in current day and NextGen aviation environments. 

We first reviewed existing metrics of SA, workload, and performance, and evaluated each metric for 
metric properties of reliability, validity, sensitivity, diagnosticity, usability, and operator acceptance. 
Based on our review of SA and SA measures, we determined that probe techniques are the most 
promising measures for evaluating the impact of NextGen changes in roles and responsibilities on 
operator SA and workload. The probe techniques have been shown to have relatively good 
psychometric properties, especially predictive validity and diagnosticity (e.g., Endsley, 1990; Endsley 
& Smolensky, 1998). With regard to existing probe techniques, we determined that real-time probe 
methods were more advantageous than freeze-probe methods because real-time probes are not 
heavily dependent on working memory and can assess SA when operators off-load information to the 
environment (Chiappe, Strybel, & Vu, in press). We also determined that some issues related to the 
existing real-time probe technique, the Situation Present Assessment Method (SPAM), needed to be 
addressed to make the method applicable for NextGen investigations of SA and workload. In 
particular, we determined that the impact of workload must be clearly identified in the use of online 
probes because workload can impact operator performance independent of SA. Relating to this point, 
we ran several studies that examined how workload and other factors affected SA measures using the 
on-line probe technique. We found: 

• Latencies to SA probes can be influenced by the response input method. Therefore, we 
streamlined the administration of probes from free-responses to multiple-choice and 
true/false or yes/no format, to reduce the added workload produced by the query task. We 
also improved the apparatus for presenting queries and collecting operator responses. 

• We determined that a standard method for developing probe queries needed to allow 
comparison of SA metrics across simulation studies. In our approach, queries were created 
beforehand in conjunction with scenario development. Subject matter experts were 
consulted, but we also created queries based on categories relevant to SA: processing level, 
time frame, and information content. After several part task simulations, we determined that 
questions should be based on information requirements. Moreover, once categories are 
established, we found that it was important to design probes so that each combination of 
categories is probed an equal number of times and the order in which information queries 
are presented must be counterbalanced across scenarios and participants. 

We ran several simulations for refining and validating our online probe technique. The technique is 
based on Durso’s SPAM technique (Durso & Dattel, 2004; see Figure 1). To ensure that response 
latency was related to SA and not workload, each query begins with a “Ready” prompt and audio 
alert. Operators were instructed to respond “yes” to the ready prompt only when they had sufficient 
time to take a SA question. If the operator, for example, a pilot, responded affirmatively to the ready 
response, the probe question was immediately presented and the pilot responded by selecting the 
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answer (typically yes/no or multiple choice). The results of our metric -development work, including a 
description of the administration and question development techniques were documented in a Manual 
for Online Probing that was submitted to our NASA Technical Monitor in January 2011 (Strybel et 
al., 2011). Results of simulations using the real-time probe technique are summarized in Section 4. 


During a simulation pilot is queried 

about situation avjareness at regular 


interva 

Pilot receives 
“Ready for 

Is 


Question” 

Time 


Prompt 

„ interval 


Pilot presses OK 

measures 


button when able 
to answer 

workload 

0) 

question 


£ 

Pilot receives 


1 — 

situation 



awareness 

Time interval 


question 

measures 
* situation 


Pilot responds 
to situation 
awareness 
question ^ 

av/areness 

' 

/ 



Time 00 secs: “Ready for ' 

Question?" 

Workload 
Time = 2 
secs 


Time 02 secs: Pilot presses 
OK. 

y 

Time 02 secs: “Will you •>. 
overtake UPS 332 in the 
next 2 minutes?” 


Situation 
Awareness 
Time = 4 secs 


Time 06 secs: Pilot 
presses “NO" 


Figure 1 . Assumptions and example ( pilot query ) of our real-time SA and workload probe 
technique. 


3. Systems Analysis of NextGen Concepts of Operation 

Analyses of the information requirements for pilot and ATC tasks under different NextGen function 
allocation and automation concepts was performed in years 1 and 2 of the NRA. Because a clear 
picture of the most likely NextGen solutions was not yet available, we surveyed the views of NASA 
scientists on the proposed NextGen solutions and determined the impact that each solution would 
have on operator roles and responsibilities, workload, and SA. Interviews with pilots and ATCs were 
also conducted to determine the impact of specific changes in roles and responsibilities on the 
information and training requirements of future operators. 

3.1 Evaluation of NextGen Concepts of Operation 

In parallel with the information requirements analyses, we also conducted a review of separation 
assurance and collision avoidance operational concepts for NextGen. The review showed that the 
concepts can be distributed along two axes: the degree to which responsibility for separation 
assurance and collision avoidance is assigned to the controllers versus the pilots, and the degree to 
which automation augments or replaces controller and pilot functions. Based on an analysis of the 
implications of these concepts to NextGen, from a human factors standpoint as well as the 
technological readiness of the concepts, it was concluded that some form of supervisory control of 
separation assurance by controllers is the most viable concept. For additional details see Table 3 and 
Dwyer and Landry (2009). 
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Table 3. Taxonomy of Separation Assurance Concepts and 
Projected Impact on Situation* 


Expected Impact Comparing NextGen Environment 
to Current Environment 



Concept 

Expected 

technological 

readiness 

ATC 

situation 

awareness 

ATC 

workload 

Pilot 

situation 

awareness 

Pilot 

workload 

Concept 

appears 

viable? 


Shared 
responsibility 
no automation 

High 

Small 

reduction 

Excessive 

Moderate 

increase 

Excessive 

No 

ATC 

auto- 

mation 

Conflict ID 
only 

Medium-high 

Neutral 

Excessive 

Neutral 

Neutral 

No 

Conflict ID 
with resolution 

tools 

Medium 

Neutral 

Very 

large 

increase 

Neutral 

Neutral 

Yes 

Conflict ID 
with resolution 

options 

Medium 

Small 

reduction 

Moderate 

increase 

Neutral 

Neutral 

Yes 

Conflict Id 
with auto- 
resolver 

Medium 

Large 

reduction 

Moderate 

reduction 

Neutral 

Neutral 

Yes 


Distributed 

control 

Low 

Very large 
reduction 

Large 

reduction 

Large 

increase 

Large 

increase 

No 


Mixed 

concepts 

Medium-low 

Moderate 

reduction 

Moderate 

increase 

Moderate 

increase 

Moderate 

increase 

Yes 


* Dwyer and Landry, 2009. 


3.2 Analysis of Information Requirements 

To support the evaluation of feasible function allocation concepts for separation assurance systems, 
and to develop a better understanding of the specific information requirements for key tasks 
(resolving conflicts, avoiding weather, and merging and spacing), air traffic controllers and 
commercial pilots were interviewed for their goals, sub-goals, and the individual and shared 
information needed to perform the tasks. For high-level goals, the two pilots interviewed were in 
general agreement in terms of their responses, even though they flew for different airlines. The two 
controllers interviewed were also in consensus. However, for the same high-level goals, the sub-goals 
of the pilots and the controllers showed more differences than similarities. For example, in the 
conflict resolution task, the effect of allocating separation responsibility to pilots (Concept 1), ATCs 
(Concept 2) and Automation (Concept 3) is shown in Table 4. Pilots based their decision for 
trajectory changes on factors such as fuel consumption, time to destination, secondary conflicts, and 
passenger comfort. The controllers, on the other hand, based their decisions on factors such as the 
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effect that the trajectory change has on the entire traffic flow, additional conflicts induced, and 
workload. Thus, while the controllers and the pilots shared common goals at the highest level, the 
controllers’ motivations are system-centric, while the pilots are aircraft-centric. This point was further 
reinforced in the group interview session in which the controllers confirmed that they rarely take into 
account the sub-goals that are important to the pilots, while the pilots indicated that they prefer the 
controllers to take into account as much as possible the sub-goals that are important to them. The key 
information requirements obtained can be used as input to ascertain which information is most 
needed when developing probe questions for measuring individual and shared SA. The knowledge 
elucidated also provided insights into the interaction among the controllers, pilots, and automation, 
and their perception of the feasibility of different concepts for separation assurance. 


Table 4. Pilot and ATC Goals and Sub-goals Across 
Function Allocation Concepts for Resolve Conflict Task* 



Concept 1 

Concept 2 

Concept 3 

PILOT 

Goals 

• Aware of conflicts near 
and far 

• Avoid secondary 
conflict 

• Avoid conflict 

• Avoid conflict 

Sub goals 

• Fuel management 

• Time management 

• Manage alt/head 

• Communicate with 
ATC 

• Fuel management 

• Passenger comfort 

• Communicate with 
ATC 

• Fuel management 

• Passenger comfort 

• Communicate with ATC 


ATC 


Goals 

• Separation 

• Separation 

• Separation 

Sub goals 

• Reduce workload 

• Reduce workload 

• Fuel levels for 


• Communicate less 

• Communicate less 

sequence priority 


• Sector traffic 

• Fuel levels for 
sequence priority 

• Sector traffic 

• Reduce sector 
workload 


* Ho, Martin, Bellissimo, and Berson, 2009. 


3.3 Implications of NextGen Roles and Responsibilities for Operator Training 

Dwyer, Gershzohn and Thorpe (2011) conducted interviews of ATCs who participated in a 
distributed simulation of potential NextGen Concepts of Operation (see Section 4.2) to assess their 
opinions on current and future training needs. Although ATC participants were initially skeptical of 
some concepts (i.e., allocated separation responsibility to the flight deck), once additional 
modifications were made to the procedures being simulated, and the ATCs were able to practice the 
concepts, they were more accepting of these concepts of operation. Dwyer et al. also asked ATCs 
about how NextGen tools will affect the training of future controllers, specifically with respect to the 
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high computing and gaming backgrounds of this generation of controllers. The majority of ATCs 
indicated that a computer gaming background might improve perceptual-motor skills, but these are 
not the critical factors determining success. In NextGen, it is likely that ATCs will be working with 
automation in the course of conducting sector- level traffic flow management tasks. Moreover, one 
ATC noted that attrition rates for trainees (which were estimated at 50%) are typically the result of 
operational errors and conducting unsafe separation operations. Perceptual-motor skills most likely 
would not contribute to these errors. 

4. Results 

4.1 Part-Task Simulations and Experiments 

A series of part-task simulations and experiments were conducted within each consortium 
organization to meet specific NRA objectives. These studies were focused on the conflict detection 
and resolution in NextGen, development of metrics for SA and workload in NextGen, application of 
the metrics to NextGen airspace environments, and the evaluation of NextGen concepts of operation 
for separation assurance. 

4. 1. 1 Conflict Detection with NextGen Flight-Deck Tools 

An airside part-task study was designed and run individually at NASA Ames’ FDDRL and three 
consortium laboratories: CSULB, CSUN, and Purdue. Flight crew responses to ground-side 
automation-derived conflict resolutions were evaluated under three flight-deck decision- aiding 
modes: automated aiding, interactive aiding, and manual resolution. In the automated aiding 
condition, pilots executed all resolutions generated by the automation. In the interactive condition, 
automation suggested a maneuver, but pilots had the choice of accepting or modifying the provided 
resolution. In the manual condition pilots generated resolutions independently. Pilots’ acceptability of 
different types of conflict resolutions provided by the automation (vertical and horizontal) at different 
ranges (near and far) from ownship was measured. Pilot SA was also determined for each of the 
flight-deck decision- aiding modes. Findings from this study were summarized in Dao et al. (2009) 
and Battiste et al. (2008). 

Conflict resolutions created either by the automation or by flight crews were safe; all resolutions 
maintained a separation distance greater than 5 nm. Crews rated approximately 30% of automated 
resolution as problematic and reported that they would seek ATC input. However, when allowed to 
modify automated resolutions with flight-deck route-planning tools, the crews wanted to consult with 
ATC on approximately 8% of the resolutions. Finally, crews reported that the decision to accept, 
reject, or modify an automated resolution is a complex and situation-dependent decision. When close 
to TOD they generally preferred to descend, but when 500 nm or more from TOD they generally 
preferred to climb. Situation awareness was measured with probe questions using a technique that 
was a combination of online- and freeze-probe methods. SA was higher in the manual- and 
interactive-aiding conditions compared with the automated condition, suggesting that pilots had 
higher SA when they were actively engaged in the conflict resolution task. It was also shown that 
pilots were more comfortable reviewing automated conflict resolutions, as well as modifying those 
resolutions before execution, compared to simply executing the resolution sent to them by the 
automated system. 

4.1.2 Metrics for Situation Awareness and Workload 

We performed several simulations to develop tools and methods for evaluating NextGen ATM 
concepts in terms of operator SA, workload, and performance, and the impact of these factors on 
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system safety and performance. Although each simulation involved both ATCs and pilots, only one 
operator role was evaluated in each study. In each study we investigated the validity, reliability, 
sensitivity and diagnosticity of workload and SA probe latencies and accuracy. 

Strybel, Vu, Kraft, and Minakata (2008) compared two methods of SA assessment with easy and hard 
self-spacing scenarios: a freeze-probe technique in which the scenario is frozen and pilots are 
administered a battery of SA questions, and an online-probe technique in which SA questions are 
administered while pilot is engaged in the primary task (flying). Performance was assessed with the 
mean and standard deviation of the indicated air speed (IAS), and number of missed ATC 
instructions. Freeze-Probe accuracy was marginally correlated with IAS variability, and online probe 
latency was significantly correlated with both IAS variability and missed ATC instructions. 

Moreover, a significant correlation between post-scenario NASA TLX and SPAM ready Latency was 
obtained, which confirmed the assumption regarding the relationship between ready latency and 
workload. A second simulation provided additional support for the diagnostic capability of online 
probes (see Section 4.2). 

The validity and diagnosticity of the online probe technique for ATCs was demonstrated in three 
part-task simulations. Strybel, Minakata, Nguyen, Pierce, and Vu (2009) examined the relative 
effectiveness of online questions designed to measure the types of processing required (recall, 
comprehension), time frame (present, future), and information/task content (sector status, conflicts, 
and command/communication). Both experienced (retired) ATCs and student ATCs were tested. The 
best types of probe questions queried operators about future conflicts and were in multiple-choice 
format. Latencies for probe questions that addressed conflicts significantly accounted for the variance 
in task performance measures (loss of separation [LOS], average vertical distance, and number of 
traffic advisories) related to safety. 

Bacon et al. (2011) examined SA, workload, and performance of retired ATCs and student ATCs in a 
simulation of mid-term NextGen airspaces. Scenarios differed in the percentage of equipped 
(DataComm, ADS-B) aircraft. ATCs could mange equipped aircraft with NextGen tools (DataComm, 
conflict probe, and a trial-planner). Conflict alerts were provided only for conflicts between two 
equipped aircraft. To obtain an overall measure of ATC performance, each scenario run was 
recorded, and ATC performance was rated by three expert controllers using a modified FAA Air 
Traffic Selection and Training (AT-SAT) Over-the-Shoulder Rating form. The modifications allowed 
for ratings regarding the use of DataComm. Experienced ATCs resolved more non-alerted conflicts 
(fewer losses of separation) than student ATCs. Experts showed lower workload (shorter ready 
latencies) and higher SA (lower SA probe latencies) than students, especially for probes relating to 
conflict queries. SA probe latency and workload (ready) latency significantly accounted for variance 
in three AT-SAT Rating Dimensions: “Maintaining Attention and Situation Awareness,” 
“Maintaining Separation” and “Overall performance.” These findings provided additional evidence 
for the validity of an online SA and workload probe method. 

Strybel et al. (2011) examined SA, workload, and performance of ATCs when the mixture of 
equipped and unequipped aircraft changed within a scenario and the change was accompanied by an 
increase in traffic, thus replicating a situation in which the percentage of equipped aircraft changes 
during an ATC’s work shift. ATC workload increased in the second half of the scenario presumably 
because of the increase in traffic. The number of LOSs between non-alerted aircraft was affected only 
by the number of conflicts. Other measures of ATC performance suggested that the strategies used 
for separating aircraft changed as a function of equipage change: When the percentage of equipped 
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aircraft increased in the second half of the scenario, the mean vertical separation between aircraft 
increased and the mean lateral separation decreased, compared with scenarios in which the 
percentage of equipped AC either decreased in the second half or remained constant. Moreover, SA 
of sector status information decreased (probe latencies increased) in the second half of scenarios in 
which the percentage of equipped aircraft decreased suggesting that SA was lower when an increase 
of unequipped aircraft was introduced into the scenario. 

In summary, our work on metric development and evaluation, specifically with the online probe 
technique, can be summarized as follows: 

• Situation awareness probe latencies are related to performance outcomes (LOS, conflict 
resolution time) that affect separation assurance for pilots and ATCs. 

• Situation awareness probe latencies for pilots are sensitive to changes in responsibility for 
separation assurance and changes in scenario difficulty brought about by weather. Workload 
probe latencies were unaffected by changes in responsibility for traffic separation. However, 
pilot workload increased throughout the descent phase, and reached a peak when queried 
before the CHRCL (outer marker) waypoint (directly before landing). 

• Situation awareness probe latencies for ATCs are sensitive to scenario manipulations such as 
equipage, and number of conflicts. Probe latencies were shown to discriminate between 
student and expert performance in mixed equipage environments, consistent with expert 
AT-SAT ratings of overall performance, maintaining separation and maintaining attention 
and SA. Probe latencies for sector status information increased when the percentage of 
equipped aircraft increased, but latencies for conflict information was unchanged, 
suggesting that only awareness of status information was reduced when the percentage of 
equipped aircraft was reduced. 

4.1.3 Alternative Approach to Situation Awareness Measurement and Validation: 

A Set Theoretic Framework 

Landry and Surakitbanharn (2011) proposed a framework for assessing the relationship between SA 
and performance based on identifying information requirements for good performance, and 
measuring operator recall of these using a freeze probe technique such as the Situation Awareness 
Global Assessment Technique (SAG AT). Moreover, once information requirements are known, it 
should be possible to identify a “mapping function” that mathematically describes the relationship 
between performance and recall of required information. If mapping functions can be found that 
relate performance to recall using SAG AT, then the value of SA for that aspect of performance can 
be validated. 

Framework: The framework assumes that certain knowledge is necessary to properly perform a task, 
and that we can model the set of all such elements of knowledge as the “target set,” as shown in 
Equation (1). 

r K = { r K,, r K 2 ,K , r K H } ^ 

The “actual” set of knowledge available to the person(s), i.e., that person’s SA, is shown in Equation 
(2). (The definition of “available” is non-trivial, but is simplified here to mean “recallable”). 

a K-{ s ,K 1 . s ,K 2 ,K, S( K„} (2) 
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A mapping function / c that relates a general set of knowledge K to some performance criterion 

relates the effect on the performance criterion of possessing K . For example, if one of the elements of 
SA is that my building is on fire, where the performance criterion is the probability I would evacuate, 
that probability of evacuating would be higher than if I did not possess that knowledge. 

We then define subsets ’KC T K and ' T K C r K such that: 

3 1C, s.t./ C| ( r K)>/ t .(.K),VKn r K-0 
3?K s.t./ c .(?K)>/ c (;K),V,-»g 

where: 

fq ( K ) - C, (5) 

Equation (3) states that there must exist some performance criterion C ; such that performance given 
the knowledge from the target set is better than if no knowledge from the target set is available. 
Equation (4) states that there exists some subset of the target set that results in as good as, if not 
better, performance than any other subset. 

Landry argues that unless (3) is true, situation awareness has no value as a construct. That is, if it is 
possible to achieve all performance criteria without the need to possess at least one particular and 
identifiable set of knowledge, SA is not a useful concept. It is possible that such a set is non- 
exclusive— there may be multiple identifiable sets, where any one of those sets is capable of 
providing the best performance; it is only necessary that these sets provide better performance than 
other sets. An important conceptual implication of this framework is that it makes the concept of SA 
falsifiable. If there are no performance criteria for which identifiable sets of knowledge improve 
performance, then SA has no validity. 

A flight simulator experiment was run in which particular aspects of performance, for which a target 
set of information and a mapping function was defined. The purpose of the experiment was, in part, 
to validate or invalidate such mapping functions. That is, this experiment was primarily designed to 
exercise the framework rather than to test the particular relationships identified. Eighteen pilot 
participants flew five profiles in which general performance data was collected, as well as their 
performance on a particular “focus event.” For each of these events, a target set of information and 
mapping function was identified. 

Half of the pilots underwent SAG AT probes during the trials, with the other half acting as a control to 
ensure that SAGAT provided neither an interruption effect nor a tip-off effect. The mapping 
functions served as a predictor of performance based on a SAGAT response to a particular question. 
SAGAT produced neither a significant tip-off nor interference effect. Overall pilot performance at the 
SAGAT queries was slightly below 50%, as was performance at the focus events. The data, in 
particular the post-hoc questionnaires, have so far suggested that the mapping functions are invalid. 
(The ability to invalidate a mapping function is useful in that it demonstrates the capability of the 
method to eliminate incorrect mapping functions.) Specifically, although the events were chosen for 
their simplicity of mapping, it appears that the mapping is more complex and that “inverse” mappings 
are largely inaccurate. Additional data analysis is being conducted to try to define accurate mapping 
functions which can then be verified in future experiments. 


(3) 

(4) 
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4.2 Distributed Air-Ground Simulations 


A series of distributed air- ground simulations were run during the period of the NRA. These involved 
simulation roles being played by two or more member organizations. Scenarios for each distributed 
simulation conducted as part of the NRA were based on sectors in Kansas City Center (ZKC) and 
Indianapolis Center (ZID), with participant pilots flying en route, arrival and approaches to Louisville 
International Airport (SDF), while avoiding weather and managing spacing, as shown in Figure 2. 



Figure 2. Simulated airspace for distributed air -ground simulations . 


4.2.1 Trajectory Oriented Operations with Limited Delegation during Convective 
Weather (TOOWiLDx) Simulation Demonstrations 

FDDRL led two simulation demonstrations in 2008 that explored flight-crew-based arrival 
management in the context of convective weather during en route and arrival operations. We 
examined whether pilot performance was affected by cockpit weather display type (current-day radar 
vs. 3D CSD). For this project, the simulation network consisted of two consortium facilities (FDDRL 
and CSULB). Participant pilots were located at FDDRL; ATC confederates were located at CSULB 
CHAAT. This demonstration was used to design two distributed simulations in 2010. 

4.2.2 Evaluation of Automated Spacing Support Tools for Interval Management 
Operations under Hazardous Weather Conditions 

Planning currently underway for future airspace management assumes that flight decks will play an 
increased role in managing the intervals between ownship and a lead aircraft when approaching an 
airport. In this study, pilots were asked to achieve a specific time-in-trail while flying an arrival into 
the SDF airport. Shortly before reaching their top of the descent, pilots were responsible for avoiding 
weather. A spacing tool calculated airspeeds designed to achieve the desired time in trail at the final 
approach fix. Pilots were exposed to four experimental conditions which varied in how strictly the 
pilots were to follow these calculated speeds, and whether these speeds had to be entered into the 
autopilot manually. Giving the pilots more discretion had little effect on the final spacing interval. 
However, when pilots were required to enter speeds into the system manually, they were less accurate 
and failed to meet altitude restrictions significantly more often. Moreover, requiring the pilots to 
manually enter speeds into the system frequently led to poorer energy management and higher 
spacing interval errors at the final approach fix, even in the conditions where pilots were instructed to 
strictly follow speed guidance. This finding was traced to poorer compliance with the automated 
speed guidance, lack of awareness of this poor compliance, and insufficient awareness of the energy 
state of the aircraft. These results suggest that some form of energy guidance may be needed to 
augment interval management. To do this, recommendations were made for integrating the spacing 
interval management automation with near-term and far-term energy management systems. While 
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these results may not always generalize to alternative spacing implementations, one should not 
assume that pilots manually closing the loop on automated commands can perform as well as a fully 
automated system. For additional details regarding this simulation see (Dao et al., 2010; Johnson et 
al., 2010). 

4.2.3 Situation Awareness with Trajectory Oriented Operations in Weather 
(SAWTOOth) 

A multi-participant distributed simulation experiment was run to examine the robustness of interval 
management during continuous descent approaches (CDAs) along the CBSKT 1 Arrival into SDF. 
Eight experimental pilots started the scenario in an en-route phase of flight and were asked to avoid 
convective weather while performing merging and spacing tasks along with a CDA into SDF. Two 
controllers managed the sectors through which the pilots flew, with one managing a sector that 
included the Top of Descent (ZKC-90), and the other managing a sector that included the merge point 
for arrival (ZID-91) into SDF. We determined the impact of changes in responsibility for separation 
assurance on the workload, SA, and performance of pilots and ATCs. Three plausible strategies for 
separation assurance were simulated and evaluated. 

Concept 1: Pilot Primary. Pilots flying equipped aircraft were given the primary 
responsibility for separation assurance between ownship and all other aircraft. Equipped 
aircraft had on-board conflict alerting, a route assessment tool (RAT), and auto-resolver tool. 
ATC was responsible for resolving conflicts between unequipped aircraft and all conflicts 
with experimental aircraft on their CDA. The human ATC was equipped with conflict 
alerting, a trial planner, and auto-resolver tool. The ground-based auto-resolver agent had no 
responsibility. 

Concept 2: ATC Primary. ATCs were given the primary responsibility for separation 
assurance. As in Concept 1, equipped aircraft had on-board conflict alerting, the RAT, and 
auto-resolver tool, but these were used for weather re-routing and proposing conflict 
resolutions. ATCs had responsibility for equipped-unequipped and unequipped-unequipped 
conflicts only, and for conflicts between any aircraft on a CDA. The autoresolver agent was 
responsible for equipped-equipped conflicts. 

Concept 3: Autoresolver Primary. In Concept 3, the autoresolver agent was responsible for 
resolving most conflicts. Aircraft were not equipped with conflict alerting conflict resolution 
capabilities, but the RAT was available to aid re-routing for weather avoidance. Pilots had no 
responsibility for separation. The autoresolver agent was responsible for conflicts between 
equipped-equipped and equipped-unequipped aircraft. The human ATC was equipped with 
conflict alerting, the trial planner, and auto-resolver tool. ATC was responsible for conflicts 
between unequipped aircraft only, and for conflicts between any aircraft on a CDA. 

In all scenarios, pilots initially engaged a spacing tool which was designed to achieve a spacing 
interval of 105 seconds from a designated lead aircraft at the final approach fix, and then re routed 
around weather. If a pilot felt that his/her weather avoidance maneuver made it unlikely that the 
spacing goal would be met, the pilot could request ATC to assign a new lead aircraft. Two additional 
variables were manipulated, weather complexity (high vs. low) and type of weather display (NextRad 
vs. 3D NextRad). High-complexity weather contained more weather cells, and covered more 
airspace. Low-complexity weather consisted of fewer weather cells. 

A set of performance measures enabled us to determine the effects of changes in responsibility for 
traffic separation on pilots and ATCs in the experiment. However, because only two ATCs were 


15 



tested in this study, the most reliable data was obtained from pilots. We evaluated the impact of each 
scenario manipulation on pilot workload, SA, spacing performance, weather avoidance, and CD A 
profile. Some of the major findings are as follows (for more information see Vu et al., 2010a, b; 
Strybel et al., 2010; Ho et al., 2010). 

• The SA and workload of pilots changed with concepts of operation. Pilots indicated that all 
concepts were workable and showed little change in workload across the three concepts. Pilots 
showed higher levels of awareness when they were responsible for maintaining separation with 
other aircraft. Although the controllers also indicated that all concepts were workable, they 
reported increased workload depending on concept of operation and sector. 

• The online probe method for assessing pilot SA was effective in detecting changes in pilot roles 
and responsibilities for traffic separation. The change in pilot tasks regarding traffic separation 
could be determined from the response latencies for questions related to traffic conflicts. These 
changes in awareness could not be attributed to workload because workload probe latencies 
were unaffected by changes in responsibility. 

• Weather perturbations can create situations in which aircraft spacing can be negatively affected 
regardless of whether the pilot faithfully followed the leading aircraft around weather. If pilots 
do not fully understand the rationale and logic of the spacing automation and its limitations, 
then trusting the automation to be able to resolve a spacing error (i.e., being ahead or behind 
schedule) to regain a lost or interrupted spacing assignment requires that pilots be highly 
confident in the automation. 


5. Conclusions 

Our multidisciplinary, research consortium of scientists and engineers from California State University 
Long Beach, San Jose State University Foundation, California State University Northridge, Purdue 
University, and The Boeing Company was successful in meeting the majority of our proposed NRA 
objectives. Because of the knowledge and experience of our members, as well as the unique approaches 
taken by them, we were able to work in parallel to identify SA and performance requirements brought 
about by anticipated changes in NextGen airspaces, develop SA and workload metrics, distributed 
simulation capabilities for evaluating NextGen concepts of operation, and assess the impacts of 
potential NextGen concepts of operations on SA, workload, and performance. 

As a result of this NRA, we have gained a greater understanding of SA and its measurement, and have 
shared our knowledge with the scientific community: During the NRA, we published 20 technical 
papers, made 21 presentations at conventions, presented 6 briefings to NASA, and prepared 5 
unpublished reports. Additional technical papers based on the NRA work were also published after the 
funding period. Our metric work improved and validated existing tools for measuring SA and 
workload, and the resulting probe tool we developed can be used by air traffic management scientists 
and engineers for assessing new concepts of operation, determining the effectiveness of automated 
tools, comparing different interface designs, and establishing training requirements. We also 
recommended a set of performance measures that will make it easier for researchers to compare results 
between experiments, simulations, and operational settings. 

Through the success of our distributed-simulation network, we have shown how to create a flexible 
and cost-effective test bed for simulations and research involving current and future airspaces. This 
network provides a mechanism for consortium members, colleagues, and students to pursue research 
on other topics in air traffic management and aviation, thus enabling them to make greater 
contributions to the field. 
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6. Publications and Presentations Citing the NRA 


Year 1 (October 2006-September 2007) 

Publications 

Battiste, V., Johnson, W. W., Johnson, N. H., Granada, S., & Dao, Q. (2007). Flight crew perspective 
on the display of 4D information for en route and arrival merging and spacing. In J. Jacko (Ed), 
Human-Computer Interaction, Part I, HC1I 2007, Lecture Notes in Computer Science 4551 
ISBN: 978-3-540-73106-1. Berlin: Springer- Verlag. 

Strybel, T. Z., Vu, K.-P. L„ Dwyer, J. P„ Kraft, J„ Ngo, T. K„ Chambers, V., & Garcia, F. P. (2007). 
Predicting perceived situation awareness of low altitude aircraft in terminal airspace using probe 
questions. In J. Jacko (Ed), Human-Computer Interaction, Part 1, HC11 2007, Lecture Notes in 
Computer Science 4550 ISBN: 978-3-540-73104-7. Berlin: Springer- Verlag. 

Presentations 

Battiste, V., Johnson, W. W., Johnson, N. H., Granada, S., & Dao, Q. (2007, July). Flight crew 

perspective on the display of 4D information for en route and arrival merging and spacing. Paper 
presented at Human Computer Interaction International 2007 Conference. Beijing, China. 

Strybel, T. Z., Vu, K.-P. L., Dwyer, J. P., Kraft, J., Ngo, T. K., Chambers, V., & Garcia, F. P. (2007, 
July). Predicting perceived situation awareness of low altitude aircraft in terminal airspace using 
probe questions. Paper presented at Human Computer Interaction International 2007 Conference. 
Beijing, China. 

Strybel, T. Z., Vu, K.-P. F., Battiste, V., & Johnson, W. W. (2007, November). Conducting 
Distributed Simulations Over the Internet. Poster accepted for presentation at the 37 th Annual 
Meeting of the Society for Computers in Psychology. Fong Beach, CA. 

Unpublished Reports 

Strybel, T. Z., Vu, K.-P. F., Battiste, V., Dao, A. Q., Dwyer, J. P., Ho, N., & Fandry S. (2007). NRA 
Year 1 Report. 

Briefings 

Strybel et al. (2006). Metrics for Operator Situation Awareness, Workload and Performance in 
Automated Separation Assurance Systems Kickoff Briefing , January, 2007. 

Strybel et al., (2007). Metrics for Operator Situation Awareness, Workload and Performance in 
Automated Separation Assurance Systems. Presented at the NASA Ames Technical Interchange 
Meeting, March, 2007 
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Year 2 (October 2007-September 2008) 

Publications 

Pierce, R., Strybel, T., & Vu, K.-P. L. (2008). Comparing situation awareness measurement 

techniques in a low fidelty air traffic control simuluation. Proceedings of the 26th International 
Congress of the Aeronautical Sciences (ICAS). Anchorage, AS. 

Pierce, R., Vu, K.-P. L., Nguyen, J., & Strybel, T. (2008). The relationship between SPAM, 
workload, and task performance on a simulated ATC task. Proceedings of the 52nd Human 
Factors and Ergonomics Society Annual Meeting. New York, NY. 

Strybel, T. Z., Vu, K.-P. L., Kraft, J., & Minakata, K. (2008). Assessing the situation awareness of 
pilots engaged in self spacing. Proceedings of the 52nd Human Factors and Ergonomics Society 
Annual Meeting. New York, NY. 

Battiste, V., Johnson, W. W., Dao, A.-Q., Brandt, S., Johnson, N., & Granada, S. (2008). 

Assessment of Flight Crew Acceptance of Automated Resolutions Suggestions and Manual 
Resolution Tools. Proceedings of the 26th International Congress of the Aeronautical Sciences 
(ICAS). Anchorage, AS. 

Presentations 

Pierce, R., Strybel, T., & Vu, K.-P. L. (2008). Comparing situation awareness measurement 
techniques in a low fidelty air traffic control simuluation. Presented at the 26th International 
Congress of the Aeronautical Sciences (ICAS). Anchorage, AS. 

Pierce, R., Vu, K.-P. L., Nguyen, J., & Strybel, T. (2008). The relationship between SPAM, 

workload, and task performance on a simulated ATC task. Presented at the 52nd Human Factors 
and Ergonomics Society Annual Meeting. New York, NY. 

Strybel, T. Z., Vu, K.-P. L., Kraft, J., & Minakata, K. (2008). Assessing the situation awareness of 
pilots engaged in self spacing. Presented at the 52ncl Human Factors and Ergonomics Society 
Annual Meeting. New York, NY. 

Battiste, V., Johnson, W. W., Dao, A.-Q., Brandt, S., Johnson, N., & Granada, S. (2008). Assessment of 
Flight Crew Acceptance of Automated Resolutions Suggestions and Manual Resolution Tools. 
Presented at the 26th International Congress of the Aeronautical Sciences (ICAS). Anchorage, AS. 

Martin, P., Ho, N. and Berson, B. (2008) Identification of shared situation awareness information 
requirements for NGATS concepts. Presented at Human Factors and NextGen; The Future of 
Aviation, Arlington TX, May 2008. 

Unpublished Reports 

Dao, A-Q & Strybel, T.Z. (2008). Demonstration of distributed simulation capabilities. Unpublished 
Report Prepared for the Consortium for the Study of Situation Awareness. 

Dwyer, J.D. & Landry, S. (2008). Roles and responsibilities conceptualizations for next generation 
air transportation system. Unpublished Report Prepared for the Consortium for the Study of 
Situation Awareness. 

Strybel, T. Z., Vu, K.-P. L., Battiste, V., Dao, A. Q., Dwyer, J. P., Ho, N., & Landry S. (2008). NRA 
Year 2 Report. 
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Briefings 

Strybel et al. Metrics for operator situation awareness, workload and performance in automated 
separation assurance systems: year 2 progress report. Presented to Todd Farley and FDDRL, 
October, 2008. 

Year 3 (October 2008-September 2009) 

Publications 

Dao, A. Q., Brandt, S. L., Battiste, V., Vu, K.-P. L., Strybel, T. Z., & Johnson, W. W. (2009). The 
impact of automation assisted aircraft separation on situation awareness. Human Interface, Part 
II, HC II 2009, Lecture Notes in Computer Science, 5618, 738-747. 

Dwyer, J. P., & Landry, S. (2009). Separation assurance and collision avoidance concepts for the 
Next Generation Air Transportation System. Human Interface, Part 11, HCII 2009, Lecture Notes 
in Computer Science, 5618, 748-757. 

Ho, N. T., Martin, P., Bellissimo, J., Berson, B. (2009). Information requirements and sharing for 
NextGen function allocation concepts. Human Interface, Part II, HCII 2009, Lecture Notes in 
Computer Science, 5618, 776-785. 

Ligda, S. V, Johnson, N., Lachter, J., & Johnson, W. W. (2009). Pilot confidence with ATC 

Automation using cockpit situation display tool in a distributed traffic management environment. 
Human Interface, Part II, HCII 2009, Lecture Notes in Computer Science, 5618, 758-775. 

Strybel, T., Minakata, K., Nguyen, K, Pierce, R., & Vu, K.-P. L. (2009). Optimizing online situation 
awareness probes in air traffic management tasks. Human Interface, Part II, HCII 2009, Lecture 
Notes in Computer Science, 5618, 845-854. 

Vu, K.-P. L., Minakata, K., Nguyen, J, Kraut, J., & Raza, H. (2009). Situation awareness and 

performance of students versus experienced air traffic controllers. Human Interface, Part II, HCII 
2009, Lecture Notes in Computer Science, 5618, 865-874. 

Presentations 

Dao, A. Q., Brandt, S. L., Battiste, V., Vu, K.-P. L., Strybel, T. Z., & Johnson, W. W. (2009). The 
impact of automation assisted aircraft separation on situation awareness. Presented at HC I 
International, San Diego, CA, July, 2009. 

Dwyer, J. P., & Landry, S. (2009). Separation assurance and collision avoidance concepts for the 
Next Generation Air Transportation System. Presented at HCI International, San Diego, CA, 

July, 2009. 

Ho, N. T., Martin, P., Bellissimo, J., Berson, B. (2009). Information requirements and sharing for 
NGATS function allocation concepts. Presented at HCI International , San Diego, CA, July, 2009. 

Ligda, S. V, Johnson, N., Lachter, J., & Johnson, W. W. (2009). Pilot confidence with ATC 

automation using cockpit situation display tool in a distributed traffic management environment. 
Presented at HCI International, San Diego, CA, July, 2009. 

Strybel, T., Minakata, K., Nguyen, K, Pierce, R., & Vu, K.-P. L. (2009). Optimizing online situation 
awareness probes in air traffic management tasks. Presented at HCI International , San Diego, 

CA, July, 2009. 

Vu, K.-P. L., Minakata, K., Nguyen, J, Kraut, J., & Raza, H. (2009). Situation awareness and 
performance of students versus experienced air traffic controllers. Presented at HCI 
International, San Diego, CA, July, 2009. 
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Unpublished Reports 

Strybel, T. Z., Vu, K.-P. L., Battiste, V., Dao, A. Q., Dwyer, J. P., Ho, N., & Landry S. (2007). NRA 
Year 3 Report. 

Briefings 

Strybel, T.Z. & Johnson, W. Situation awareness with trajectory oriented operations in weather 
(SAWTOOth): Design Review. Presented to NASA Ames March, 2009. 

Year 4 (October 2010-January 2011) 

Publications 

Dao, A-Q.V., Lachter, J., Battiste, V., Brandt, S.L., Vu, K-P.L., Strybel, T.Z., Ho, N., Martin, P., & 
Johnson, W.W. (2010). Automated spacing support tools for interval management operations 
during continuous descent approaches. In Proceedings of the 54 th Annual Meeting of the Human 
Factors & Ergonomics Society, San Francisco, CA: Human Factors Society. 

Ho, N., Johnson, W., Martin, P., Lachter, J., Dao, A.-Q., Brandt, S & Battiste, V. (2010). Pilot 
response to off-nominal conditions in merging and spacing operation. Proceedings of the 29th 
Digital Avionics Systems Conference. Salt Lake City, Utah: IEEE. 

Johnson, W., Ho, N., Battiste, V., Vu, K.P. L., Lachter, J., Ligda, S., Dao, A.-Q., & Martin, P. (2010). 
Management of continuous descent approach during interval management operation. Proceedings 
of the 29th Digital Avionics Systems Conference . Salt Lake City, Utah: IEEE. 

Ligda, S.V., Dao, A-Q.V., Strybel, T.Z., Vu, K-P.L., Battiste, V., & Johnson, W.W. (2010). Impact of 
conflict avoidance responsibility allocation on pilot workload in a distributed air traffic 
management system. In Proceedings of the 54 th Annual Meeting of the Human Factors & 
Ergonomics Society, San Francisco, CA: Human Factors Society. 

Strybel, T.Z., Vu, K-P.L., Bacon, L.P., Kraut, J., Battiste, V., & Johnson, W. (2010). Diagnosticity of 
an online query technique for measuring pilot situation awareness in NextGen. In Proceedings of 
the 29 th Digital Avionics Systems Conference, Salt Lake City, UT. 

Vu, K-P.L., Strybel, T.Z., Kraut, J., Bacon, L.P., Minakata, K., Nguyen, J., Rotterman, A., Battiste, 
V., & Johnson, W. (2010). Pilot and controller workload and situation awareness with three 
traffic management concepts. In Proceedings of the 29 th Digital Avionics Systems Conference, 

Salt Lake City, UT. 

Vu, K-P. L., Strybel, T. Z., Battiste, V. & Johnson, W. (in press). Factors Influencing the Decisions 
and Actions of Pilots and Controllers in Three Plausible NextGen Environments. In R. W. Proctor 
and S. Nof. Y. Yih, (Eds.) Cultural Factors in Decision Making and Action. CRC Press. 

Presentations 

Dao, A-Q.V., Lachter, J., Battiste, V., Brandt, S.L., Vu, K-P.L., Strybel, T.Z., Ho, N., Martin, P., & 
Johnson, W.W. (2010). Automated spacing support tools for interval management operations 
during continuous descent approaches. Presented at the 54 th Annual Meeting of the Human 
Factors & Ergonomics Society, September 2010, San Francisco, CA. 

Ho, N., Johnson, W., Martin, P., Lachter, J., Dao, A.-Q., Brandt, S & Battiste, V. (2010). Pilot 
response to off-nominal conditions in merging and spacing operation. Presented at the 29th 
Digital Avionics Systems Conference, October, 2010, Salt Lake City, Utah. 


20 



Johnson, W., Ho, N., Battiste, V., Vu, K.P. L., Lachter, J., Ligda, S., Dao, A.-Q., & Martin, P. (2010). 
Management of continuous descent approach during interval management operation. Presented at 
the 29th Digital Avionics Systems Conference, October, 2010, Salt Lake City, Utah. 

Ligda, S.V., Dao, A-Q.V., Strybel, T.Z., Vu, K-P.L., Battiste, V., & Johnson, W.W. (2010). Impact of 
conflict avoidance responsibility allocation on pilot workload in a distributed air traffic 
management system. Presented at the 54 th Annual Meeting of the Human Factors & Ergonomics 
Society, September 2010, San Francisco, CA. 

Strybel, T.Z., Vu, K-P.L., Bacon, L.P., Kraut, J., Battiste, V., & Johnson, W. (2010). Diagnosticity of 
an online query technique for measuring pilot situation awareness in NextGen. Presented at the 
29th Digital Avionics Systems Conference, October, 2010, Salt Lake City, Utah, 

Vu, K-P.L., Strybel, T.Z., Kraut, J., Bacon, L.P., Minakata, K., Nguyen, J., Rotterman, A., Battiste, 
V., & Johnson, W. (2010). Pilot and controller workload and situation awareness with three 
traffic management concepts. Presented at the 29th Digital Avionics Systems Conference, 

October, 2010, Salt Lake City, Utah, 

Vu, K-P. L., Strybel, T. Z., Battiste, V. & Johnson, W. (2010). Factors Influencing the Decisions and 
Actions of Pilots and Controllers in Three Plausible NextGen Environments. Presented at 
Cultural Factors in Decision Making and Action, November 2010, Lafayette, Indiana. 

Unpublished Reports 

Dwyer, J. P., Gershzohn, G., & Thorpe, E.M. Review of Air Traffic Controller and Pilot Training for 
The Sawtooth II Airborne Self-Separation, Continuous Descent Arrival NextGen Air Traffic 
Management Concept Simulation. Presented to NRA Consortium January 2011 . 

Strybel, T.Z., Vu, K.P.L., Chiappe, D. Battiste, V. & Dao, A-Q. V. Metrics for Situation Awareness, 
Workload, and Performance: Manual for Online Probe Administration. Presented to Walt 
Johnson NRA Technical Monitor January, 2011 . 


Post-Award Period Publications Citing Work Supported by the NRA 

Bacon, L.P., Strybel, T.Z. Vu, K.-P.L., Kraut, J.M., Nguyen, J.H., Battiste, V., & Johnson, W. W. 
(2011) Situation awareness, workload and performance in midterm NextGen: Effect of variations 
in aircraft equipage levels between scenarios. Proceedings of the 2011 International Symposium 
on Aviation Psychology (pp. 32-37), Dayton, OH. 

Kiken, A., Rorie, R.C, Bacon, L. P., Billinghurst, S., Kraut, J.M., Strybel, T. Z., Vu, K.-P. L., & 
Battiste, V. (2011). Effect of ATC training with NextGen tools and online situation awareness 
and workload probes on operator performance. In G Salvendy & M. J. Smith (Eds.), Human 
Interface, Part 11, HCII 2011 , Lecture Notes in Computer Science, 6772 , 483-492. 

Kraut, J., Kiken, A., Billinghurst, S., Morgan, C., Strybel, T. Z., Chiappe, D. L., & Vu, K.-P. L. 
(2011) Effects of data communications failure on air traffic controller sector management 
effectiveness, situation awareness and workload. In G. Salvendy and M. J. Smith (Eds.), Human 
Interface, Part II, HCII 2011 , Lecture Notes in Computer Science, 6772, 493-499. 

Landry, S., J., & Surakitbanharn, C. (2011). A formalism for assessing the situation awareness of 
pilots. In G. Salvendy and M. J. Smith (Ed.s), Human Interface, Part 11, HCII 2011, Lecture 
Notes in Computer Science, 6781 , 572-581. 


21 



Strybel, T. Z., Vu, K. L., Bacon, L. P., Kiken, A., Billinghurst, S., Rorie, R. C., Morgan, C., &, 

Battiste, V. (2011) Situation awareness, workload and performance in midterm NextGen: Effect of 
dynamic variations in aircraft equipage levels. Proceedings of the 2011 International Symposium 
on Aviation Psychology , (pp 118-123). Dayton, OH. 

Strybel, T. Z., Vu, K.-P. L, Chiappe, D., Battiste, V., Johnson, W., & Dao, Q., V. (2011). Metrics for 
situations awareness, workload, and performance : Manual for online probe administration. 
(Report no. NAS A/TM-201 1-215991). Moffett Field, CA: NASA Ames Research Center. 


7. References 

Chiappe D. L., Strybel, T. Z., & Vu, K.-P.L. (in press). Mechanisms for the acquisition of situations 
awareness in situated agents. Theoretical Issues in Ergonomics Sciences, DOI: 
10.1080/1463922X.201 1 .61 1267 

Durso, F. T., & Dattel, A. (2004). SPAM: The real-time assessment of SA. In S. Banbury & S. 

Trembley (Eds.), A cognitive approach to situation awareness: Theory, measures and application 
(pp. 137-154). New York: Aldershot. 

Endsley, M. R. (1990). Measurement of situation awareness in dynamic systems. Human Factors, 37, 
65-84. 

Endsley, M. R., & Smolensky, M. (1998). Situation awareness in air traffic control: The Picture. In 
M. Smolensky & E. Stein (Eds.) Human factors in air traffic control (pp. 115-154). New York: 
Academic Press. 


22 



23 



Report Documentation Page 

Form Approved 
OMB No. 0704-0188 

The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, 
gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this 
collection of information, including suggestions for reducing this burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and 
Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person 
shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. 

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 

1. REPORT DATE (DD-MM-YY) 

12-15-11 

2. REPORT TYPE 

Technical Memorandum 

3. DATES COVERED (From - To) 


4. TITLE AND SUBTITLE 


Metrics for Operator Situation Awareness, Workload, and 
Performance in Automated Separation Assurance Systems 


5a. CONTRACT NUMBER 


5b. GRANT NUMBER 


5c. PROGRAM ELEMENT NUMBER 


6. AUTHOR(S) 

Thomas Z. Strybel, Kim-Phuong L. Vu, Vernol Battiste, Arik- 
Quang Dao, John P. Dwyer, Steven Landry, Walter Johnson, 
Nhut Ho 


5d. PROJECT NUMBER 


5e. TASK NUMBER 


5f. WORK UNIT NUMBER 

411931.02.51.01.08 


7. PERFORMING ORGANIZATION NAME(S) AND ADDRESSES(ES) 

NASA Ames Research Center 
Moffett Field, California 94035-1000 


8. PERFORMING ORGANIZATION 
REPORT NUMBER 

TH-091 


9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 

National Aeronautics and Space Administration 
Washington, DC 20546-0001 


10. SPONSORING/MONITOR’S ACRONYM(S) 

NASA 

1.1. 1.1.1 11. SPONSORING/MONITORING 

REPORT NUMBER 

NAS A/TM-201 1-215997 


12. DISTRIBUTION/AVAILABILITY STATEMENT 

Unclassified— Unlimited 
Subject Category: 3 


Availability: NASA CASI (301) 621-0390 


Distribution: Nonstandard 


13. SUPPLEMENTARY NOTES 

Point of Contact: Walter Johnson, NASA Ames Research Center, MS 262-4, Moffett Field, CA 94037; 
(650) 604-3667. 


14. ABSTRACT 

A research consortium of scientists and engineers from California State Univ. Long Beach, San Jose State Univ. 
Foundation, California State Univ. Northridge, Purdue Univ., and the Boeing Company was assembled to evaluate the 
impact of changes in roles and responsibilities and new automated technologies, being introduced in the Next Generation 
Air Transportation System, on operator situation awareness and workload. To meet these goals, consortium members 
performed systems analyses of NextGen concepts and airspace scenarios and concurrently evaluated SA, workload, and 
performance measures to assess their appropriateness for evaluations of NextGen concepts and tools. The following 
activities and accomplishments were supported by the NRA: a distributed simulation, metric development, systems 
analysis, part-task simulations, and large-scale simulations. As a result of this NRA, we have gained a greater 
understanding of situation awareness and its measurement, and have shared our knowledge with the scientific 
community. This network provides a mechanism for consortium members, colleagues, and students to pursue research 
on other topics in air traffic management and aviation, thus enabling them to make greater contributions to the field. 


15. SUBJECT TERMS 

situation awareness; workload; NextGen 


16. SECURITY CLASSIFICATION OF: 


a. REPORT 

U 


b. ABSTRACT 

u 


c. THIS PAGE 

U 


17. LIMITATION 
OF 

ABSTRACT 

UU 


18. NUMBER 
OF 

PAGES 

32 


19a. NAME OF RESPONSIBLE PERSON 

STI Help Desk at email: 
help @ sti .nasa.gov 


19b. TELEPHONE NUMBER (Include area code) 

(443) 757-5802 


Standard Form 298 (Rev. 8-98) 




